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■ Motivation and goals 

■ The big picture 

■ Blender and its Game Engine 

■ CLAM and its 3D-audio engine 

■ The systems communication via SpatDIF 

■ Game demo (video) 

■ Conclusions and future work 
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Motivation 


■ Blender has produced impressive 
demonstrators: 


■ Elephant Dreams and Big Buck Bunny 3D movies 

■ Yo Frankie! 3D game 

■ We have recently developed a 3D-audio 
platform based on the CLAM framework 


■ Room acoustics simulation, ambisonics, binaural, 
Vector Based Amplitude Panning, etc. 

■ We faced the need for powerful 3D geometric 
tools to drive the 3D-audio rendering 
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Goals 



■ Take advantadge of Blender! 

■ Experiment with 3D audio games 

■ Set up an experimental platform to do 3D 
audio scene rendering 

■ Start with two decoupled systems 

■ Using OSC to link the geometric scene with the 
audio rendering -► lucky strike: SpatDIF protocol 

■ Allow changing the exhibition system and rendering 
algorithms in CLAM 

■ Allow changing the graphics engine and 3D scenes 
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The big picture 



Blender (3D Scene/interaction) 

Yo Frankie! 


Hacks 

(OSC/SpatDIF 
python scripts 
senders) 


osc / 

SpatDIF 
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CLAM (Audio Render) 
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Blender and its Game Engine 


Authoring 


18 ;| ^ File Add Timeline Game Render Help | ?|SR:2-Model |x] [ f1 SCE:frank_logic |X~| V ' www.blender.org 248.2 Ve:1748 | Fa:1785 | Ob:3-1 | La:0 | Mem:25.77M (9.34M) | Time: | MeshMomo 



def getTypeOfObject(controller) 

dictTypes={"moving_source": “source", 

"fixed_source": "source", 

"listener": "listener" } 
for sensorName in dictTypes.keys(): 

sensor=controller.getSensor(sensorName) 

if sensor.triggered and sensor.positive: 
return dictTypestsensorName] 
return False 

# use OSC client module for python - by Stefan Kersten 
home=getenv( "HOME") 

pathToOSCList=[". ./osc/oscpython" , "*s/src/liblo" % home, “*s/acusticc 
configured=0 

for testpath in pathToOSCList: 

if Blender,sys.exists(testpath+"/OSC. py”): 
path,append(test path) 
from OSC import Message 
configured=l 
break 

if configured==8: 

print "Can't found OSC.py. Aborting." 

def sendObjectValue(objectName,typeName,typeValue,value,port): 

message="/SpatDIF/*s/Ss/xyz/#s" % (typeName,object Name,typeValue) 

# Message(message,value).sendto("nameofserver.com",port) 

| Message;message,value).sendlocal(port) 




def main(): 

# print "main de BlenderOSCSender" 
typename=None 

controller=QameLogic,getCurrentController() 
ob ject controller, get Owner () 

# print object.name 
location=object.getPosition() 

# print location 
ori=object.orientation 

# print ori 

mat r=B1ender,Mathutils.Matrix(ori[0],ori[l],ori[2]) 
Text Edit Format |Q| B )a-= |ab |^] | ?|EngineOSCSender.p| 



Barcelona 

Media 


LAC 09 - April 16 th Parma 

























































Blender's Game Engine 


■ Real-time graphics rendering, physics 
simulations 

■ Interactivity defined via blocks design 

■ (Easily) extensible via Python scripting 

■ GameLogic Python module 

■ Game logic defined by: 

■ Sensors: generate events 

■ Controllers', combine events and trigger actuators or 
Python scripts 

_ ■ Actuators', do actions to interact with the scene 
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Blender's Game Engine 
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■ A ship object sensor detects a contact with an 
asteroid object 

■ The controller receives the event and, since its 
state is ’’whithout shield” it passes the event to the 
attached actuators. 

■ The actuator receives the event and applies a 



















































































The ”Yo Frankie!” open game 



Hwww.blender.org 243.2 Ve:141301 | Fa:130605 | Ob:167-1 I La:2 
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■ Yo Frankie! Game 
logic is defined like 
this 

■ And stored in binary 
format 

■ This is too 
entangled to 
maintain! 

■ We expect this to 
improve in the 
future 
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CLAM, the audio framework 


^L/IM 

■ CLAM stands for C++ Library for Audio and 
Music 

■ Started 8 years ago at Universitat Pompeu 
Fabra / MTG, 

■ Now a comunity project, mainly used and 
supported by BarcelonaMedia 

■ BTW, new URL: http://clam-project.org 
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Some CLAM features 


■ Supports different & extensible token types 
(e.g. Samples, spectrums, features) 

■ Processing networks are multirate (e.g. 
processing A runs 3 times for each run of B) 
-► Synchronous Dataflow scheduling 

■ Multiple audio backends: Jack, Portaudio, 
LADSPA, VST 


■ Rapid prototyping (via CLAM NetworkEditor 
and QtDesigner) 



Offline operation (combined with scripting) 


(22) Barcelona 
Media 
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CLAM prototyping 
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CLAM's processings 


actor name 



class MyProcessing : CLAM::Processing 

C 

InPort<TokenType> .input 1;. 

InPort <TokenType > .input2; 

OutPort<TokenType> .output; 


state (data) 


parameters 



input stream output stream 



public: 

Adder () : _ input 1C " Input u 1 ,f >thi s ) , _ input 2 (" Input u 2 " , this), 

_outputC ” Output", this) 

C 

} 

void Do O 

C 


} 


.output.produce( _input 1.consumeO + _input2.consume() ); 






























CLAM's 3D-audio engine 


■ Audio objects belonging to a scene are 
encoded into a convenient format 


■ Audio objects and listener are then animated via 
SpatDIF messages 

■ Audio objects can be synthesized via a sampler 
triggered by SpatDIF messages, or linked to audio 
streams 


■ The audio is then rendered and decoded to 
the choosen exhibition system 

■ Direct-sound and reverb (if exists) are treated 
differently 
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Audio objects synthesis 


■ CLAM processing able to trigger sounds in 
multiple layers/voices, for each audio object 
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3D-audio rendering & decoding 


■ Direct-Sound 


■ Binaural, via a the best-matching HRTF filter 

■ (Any) Multi-loudspaker setup, using Vector Base 
Amplitude Panning (VBAP) 

■ Reverb (optional) 


■ Simulated with a ray-tracing algorithm (developed 
at BarcelonaMedia but not open-source) 

■ But the ’’navigation” though impulse-responses is 
open-source 

■ Encoded into Ambisonics (typically lrst order) 



Decoded into binaural (using many virtual 
loudspeakers) and into multi-loudspeakers. 

rr' I r\ r\ " 
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CLAM's 3D-audio engine 


■ Demo ray-tracing 

■ Next slides: rendering and decoding networks 

■ Binaural rendering and decoding 

■ Bformat (1 st order Ambisonics) decoding to 
surround 
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■ Partitioned 
Convolution 
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Decoding options 



■ Binaural (based on HRTFs) with head-tracking 
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More decoding options 
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15 speakers in our lab (4+6+4+1) 
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More decoding options 
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■ 5.1, 22.2, etc 

■ Adding new loudspeakers 
setups is very simple. All it 
takes is a configuration file 
with loudspeaker-positions 


Q 3 H 
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Direct-sound + reverb 
simulation in binaural 
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Impulse Response Loader Vx 
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■ Bformat to surround 5.0 
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The audio render user interface 


■ All the networks are behind the courtains 
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I The SpatDIF protocol 


■ Spatial Sound Description Interchange Format 

■ The definition of the format is work in progress 

■ Now it is the right moment to give feedback 

■ It is built on top of Open Sound Control (OSC). 
Which is good! 

■ The Blender-CLAM system uses a subset of 
SpatDIF and some extensions by our own 
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SpatDIF messages examples 


real time 



■ /SpatDIF/source/n/xyz (x-axis y-axis y-axis; floats) 

■ /SpatDIF/source/n/aed (azimuth elevation distance; floats) 

■ /SpatDIF/source/n/aer (azimuth elevation roll; floats) 

■ /SpatDIF/source/n/emissionPattern (cardioid, omni, etc) 

(22) Barcelona 
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SpatDIF extended messages 


■ /SpatDIF/source/n/sampler/addLayer (name ; string) 

■ /SpatDIF/source/n/sampler/name/setBuffer (audio file ; string) 

■ /SpatDIF/source/n/sampler/name/setLoop (bool) 

■ /SpatDIF/source/n/sampler/name/play (start|stop ; bool) 
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Video 
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demo (in binaural) 
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Conclusions 


■ This work is fun (and we get paid for it!) 

■ We are learning a lot along the way. Recent 
improvements: 

■ Segregation of direct sound and reverb (VBAP and 
Ambisonics) 

■ Binaural with head-tracking makes a huge 
difference 

■ In postproduction, mixing 10 tracks of 15-channels 
(VBAP) files in Ardour does not work (HD not so 
fast) 

Use VBAP decoding plugins with automations instead 
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Conclusions: other given uses 


■ Beyond Blender's Game Engine 

■ Real-time scene manipulation: Link CLAM's audio 
scene rendering with Blender (not GE), to hear the 
scene being manipulated 

■ Using motion trackers 

■ High quality (ray-tracing) offline rendering, using 
CLAM's offline network player and Python to 
stablish parameters 

■ Export Blender's scene into an Ardour's session 
(with spacialization plugins with animated controls) 

■ Several 3D audio movies segments have been produced 


(22) Barcelona 
Media 
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Conclusions: other given uses 


* ardour session - Ardour 
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Future Work 


■ Improved scene/samplers setup via SpatDIF 

■ Better Ambisonics decoder for non-regular 
setups, shelf filtering and higher order 

■ Non-punctual audio objects (e.g. a river) 

■ Encapsulate 3D audio functionality in a well 
defined library (ala openAL) 

■ Long term: incorporate geometric metaphors 
into DAWs 

■ Simplified, customized set of Blender's features 

■ Production independent of the exhibition system 
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Thanks! 


Questions? 


(You are welcomed to the CLAM workshop. 

Tomorrow at 15h) 
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Following slides not used 
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The devil is in the details 
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